{
 "nbformat": 4,
 "nbformat_minor": 0,
 "metadata": {
  "colab": {
   "provenance": [],
   "include_colab_link": true
  },
  "kernelspec": {
   "name": "python3",
   "language": "python",
   "display_name": "Python 3 (ipykernel)"
  },
  "language_info": {
   "name": "python"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "view-in-github",
    "colab_type": "text"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/LC1332/Chat-Haruhi-Suzumiya/blob/main/kyon_generator/test_kyon_generator.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Chat凉宫春日 Chat-Haruhi-Suzumiya\n",
    "\n",
    "[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)]()\n",
    "[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)]()\n",
    "[![Huggingface Gradio](https://img.shields.io/static/v1?label=Demo&message=Huggingface%20Gradio&color=orange)](https://huggingface.co/spaces/silk-road/ChatHaruhi)\n",
    "\n",
    "\n",
    "本脚本是李鲁鲁开发的KyonGenerator的测试工具，后者由\n",
    "\n",
    "李鲁鲁，闫晨曦, 包同学, 睡觉鱼等共同开发\n",
    "\n",
    "---\n",
    "\n",
    "**Chat凉宫春日**是模仿凉宫春日等一系列动漫人物，使用近似语气、个性和剧情聊天的语言模型，\n",
    "\n",
    "<details>\n",
    "  <summary> 本项目由李鲁鲁，冷子昂，闫晨曦，封小洋，scixing，沈骏一，Aria Fei, 米唯实, 吴平宇, 贾曜恺等开发。 </summary>\n",
    "\n",
    "李鲁鲁发起了项目，并完成了最早的版本，在多个微信群实现了测试。完成了训练数据自动生成对话部分，设计了整体的路线，并撰写报告。\n",
    "\n",
    "冷子昂负责了每一个阶段的Gradio开发，以及每个部分的功能整合和架构设计。\n",
    "\n",
    "闫晨曦一开始将李鲁鲁的notebook重构为app.py，参与了WebUI的embedding部分重构整合。\n",
    "\n",
    "封小洋进行了中文转日文模型的选型，完成了针对台词抽取图片的工具。整合了声纹识别。即将继续参加台本工具的开发。\n",
    "\n",
    "scixing实践了VITS语音，完成了台词对应的语音抽取，实现了第一个版本的声纹分类。\n",
    "\n",
    "沈骏一实现了使用ChatGLM2 finetune实验\n",
    "\n",
    "Aria(Yaying Fei)对接了whisper到台本工具。即将继续参加台本工具的开发。\n",
    "\n",
    "米唯实实现了Chat哆啦A梦的分支版本\n",
    "\n",
    "吴平宇部署了ChatGLM2的训练程序，并提供了训练的计算资源。\n",
    "\n",
    "贾曜恺开发了一个Vue前端实现方案\n",
    "\n",
    "</details>"
   ],
   "metadata": {
    "id": "xQqfMSu5FwUg"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "!git clone https://github.com/LC1332/Chat-Haruhi-Suzumiya"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iATX-_amFs0n",
    "outputId": "15c3149a-aaec-4d08-aa26-d836d25dd427"
   },
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Cloning into 'Chat-Haruhi-Suzumiya'...\n",
      "remote: Enumerating objects: 12963, done.\u001B[K\n",
      "remote: Counting objects: 100% (3081/3081), done.\u001B[K\n",
      "remote: Compressing objects: 100% (2943/2943), done.\u001B[K\n",
      "remote: Total 12963 (delta 132), reused 3013 (delta 121), pack-reused 9882\u001B[K\n",
      "Receiving objects: 100% (12963/12963), 158.65 MiB | 14.17 MiB/s, done.\n",
      "Resolving deltas: 100% (2102/2102), done.\n",
      "Updating files: 100% (8214/8214), done.\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "!pip install -q openai langchain"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0SytUs-oI4Hd",
    "outputId": "cfebc2eb-7b8a-4b67-8879-ed0ceca17db5"
   },
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m73.6/73.6 kB\u001B[0m \u001B[31m1.6 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m1.4/1.4 MB\u001B[0m \u001B[31m21.2 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m90.0/90.0 kB\u001B[0m \u001B[31m9.5 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m49.4/49.4 kB\u001B[0m \u001B[31m5.2 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25h"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "!pip install -q tiktoken transformers jsonlines"
   ],
   "metadata": {
    "id": "m9ykx-hICe8l",
    "outputId": "4e88b4a6-a6c1-4776-a33f-79d878748b1a",
    "colab": {
     "base_uri": "https://localhost:8080/"
    }
   },
   "execution_count": 12,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\u001B[?25l     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m0.0/1.7 MB\u001B[0m \u001B[31m?\u001B[0m eta \u001B[36m-:--:--\u001B[0m\r\u001B[2K     \u001B[91m━\u001B[0m\u001B[91m╸\u001B[0m\u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m0.1/1.7 MB\u001B[0m \u001B[31m1.9 MB/s\u001B[0m eta \u001B[36m0:00:01\u001B[0m\r\u001B[2K     \u001B[91m━━━━━━━━\u001B[0m\u001B[90m╺\u001B[0m\u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m0.4/1.7 MB\u001B[0m \u001B[31m5.1 MB/s\u001B[0m eta \u001B[36m0:00:01\u001B[0m\r\u001B[2K     \u001B[91m━━━━━━━━━━━━━━━\u001B[0m\u001B[90m╺\u001B[0m\u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m0.7/1.7 MB\u001B[0m \u001B[31m6.2 MB/s\u001B[0m eta \u001B[36m0:00:01\u001B[0m\r\u001B[2K     \u001B[91m━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m\u001B[91m╸\u001B[0m\u001B[90m━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m1.0/1.7 MB\u001B[0m \u001B[31m7.3 MB/s\u001B[0m eta \u001B[36m0:00:01\u001B[0m\r\u001B[2K     \u001B[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m\u001B[90m╺\u001B[0m\u001B[90m━━━━━━\u001B[0m \u001B[32m1.4/1.7 MB\u001B[0m \u001B[31m8.1 MB/s\u001B[0m eta \u001B[36m0:00:01\u001B[0m\r\u001B[2K     \u001B[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m\u001B[91m╸\u001B[0m \u001B[32m1.7/1.7 MB\u001B[0m \u001B[31m8.6 MB/s\u001B[0m eta \u001B[36m0:00:01\u001B[0m\r\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m1.7/1.7 MB\u001B[0m \u001B[31m7.8 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25h"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "# should be remove\n",
    "!pip install -q gradio"
   ],
   "metadata": {
    "id": "Tr-uW85pCuXZ",
    "outputId": "f47648ab-e79e-4df0-b024-7d27bbb73edd",
    "colab": {
     "base_uri": "https://localhost:8080/"
    }
   },
   "execution_count": 14,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m19.9/19.9 MB\u001B[0m \u001B[31m32.7 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m65.8/65.8 kB\u001B[0m \u001B[31m6.0 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25h  Preparing metadata (setup.py) ... \u001B[?25l\u001B[?25hdone\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m294.2/294.2 kB\u001B[0m \u001B[31m20.7 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m75.4/75.4 kB\u001B[0m \u001B[31m7.3 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m268.8/268.8 kB\u001B[0m \u001B[31m20.9 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m50.5/50.5 kB\u001B[0m \u001B[31m4.9 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m138.7/138.7 kB\u001B[0m \u001B[31m13.4 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m45.7/45.7 kB\u001B[0m \u001B[31m4.9 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m59.5/59.5 kB\u001B[0m \u001B[31m6.0 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m129.9/129.9 kB\u001B[0m \u001B[31m12.7 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m50.4/50.4 kB\u001B[0m \u001B[31m5.7 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m46.5/46.5 kB\u001B[0m \u001B[31m5.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m43.7/43.7 kB\u001B[0m \u001B[31m4.0 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m41.0/41.0 kB\u001B[0m \u001B[31m3.9 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m41.0/41.0 kB\u001B[0m \u001B[31m4.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m87.5/87.5 kB\u001B[0m \u001B[31m9.0 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m84.5/84.5 kB\u001B[0m \u001B[31m9.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m58.3/58.3 kB\u001B[0m \u001B[31m6.4 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m67.0/67.0 kB\u001B[0m \u001B[31m7.8 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m74.5/74.5 kB\u001B[0m \u001B[31m8.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25h  Building wheel for ffmpy (setup.py) ... \u001B[?25l\u001B[?25hdone\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZWCXGjG-FJdq",
    "outputId": "496389c0-5dba-426a-ce92-de95dd05e055"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "/content/Chat-Haruhi-Suzumiya/kyon_generator\n",
      "chat2dialogue.py\t   generate_character.py\n",
      "ChatGPT_for_generation.py  story2chat.py\n",
      "chatlog2dialogue.ipynb\t   synthesis_chat_method_foo.py\n",
      "chatlog2dialogue.py\t   synthesis_chat.py\n",
      "data\t\t\t   test_kyon_generator.ipynb\n",
      "dialogue2chat.py\t   train.py\n",
      "dialogue2embedding.py\n"
     ]
    }
   ],
   "source": [
    "%cd /content/Chat-Haruhi-Suzumiya/kyon_generator/\n",
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "!ls data"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5ZVbJHRUGhTK",
    "outputId": "62987bd1-ba10-4080-bacd-0439af4a2530"
   },
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Haruhi-ContinuousGenerate_Dialogues.jsonl  test_query_chat_Hermione.jsonl\n",
      "Haruhi_first_merge_res.jsonl\t\t   test_query_chat_liyunlong.jsonl\n",
      "Haruhi-Lulu_Dialogues.jsonl\t\t   test_query_chat_Malfoy.jsonl\n",
      "test_query_chat_Haruhi.jsonl\t\t   test_query_chat_weixiaobao.jsonl\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "!mkdir output"
   ],
   "metadata": {
    "id": "nTgRVhgcG39G"
   },
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## gradio前端记录的历史数据到dialogue数据\n",
    "chatlog2dialogue.py gradio前端记录的历史数据到dialogue数据的转化\n",
    "\n"
   ],
   "metadata": {
    "id": "QkKog1XUGdEm"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "!python chatlog2dialogue.py -input ./data/Haruhi-Lulu_Dialogues.jsonl\\\n",
    "                           -output ./output/Haruhi_Dialogues_from_history.jsonl"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LZmmLHFGGUgZ",
    "outputId": "8abaa541-9205-472a-ba8a-d4dc6c71720e"
   },
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "未实现"
   ],
   "metadata": {
    "id": "En_FJ9gtl0V9"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## story2chat 台本数据到一句话数据的转换\n",
    "\n",
    "Haruhi为例的话语料在\n",
    "\n",
    "/content/Chat-Haruhi-Suzumiya/characters/haruhi/texts"
   ],
   "metadata": {
    "id": "rTAg037YHTf-"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "!python story2chat.py -story_folder \"/content/Chat-Haruhi-Suzumiya/characters/haruhi/texts\" \\\n",
    "                     -output ./output/chat_from_story.jsonl \\\n",
    "                     -role \"春日\" \\\n",
    "                     -other_names 凉宫 凉宫春日"
   ],
   "metadata": {
    "id": "tJq8CswXHMar"
   },
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "!python story2chat.py -story_folder \"/content/Chat-Haruhi-Suzumiya/characters/weixiaobao/texts\" \\\n",
    "                     -output ./output/chat_from_story_weixiaobao.jsonl \\\n",
    "                     -role \"韦小宝\" \\\n",
    "                     -other_names 小宝"
   ],
   "metadata": {
    "id": "qxQqcl2tmAaQ"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "!python story2chat.py -story_folder \"/content/Chat-Haruhi-Suzumiya/characters/Hermione/texts\" \\\n",
    "                     -output ./output/chat_from_story_Hermione.jsonl \\\n",
    "                     -role \"Hermione\""
   ],
   "metadata": {
    "id": "tRu8EQiJmjch"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "测试通过 implemented by 冷月"
   ],
   "metadata": {
    "id": "Ykw3uZLzmyxV"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## dialogue2chat 将连续对话批量抽取为一句话"
   ],
   "metadata": {
    "id": "fkuR83edH7am"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "!python dialogue2chat.py -input ./data/Haruhi-Lulu_Dialogues.jsonl\\\n",
    "                     -output ./output/chat_from_dialogue.jsonl\\\n",
    "                     -role \"春日\"\\\n",
    "                     -other_names 凉宫 凉宫春日"
   ],
   "metadata": {
    "id": "zwZgBFPxH8dE"
   },
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "测试通过"
   ],
   "metadata": {
    "id": "bRmSkEmAm5tc"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## synthesis_chats 将一些第一句话的文件，增广为更多的第一句话"
   ],
   "metadata": {
    "id": "y6ykfthzIfOP"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "这里需要设置OpenAI Key"
   ],
   "metadata": {
    "id": "MZApS5ubJE-L"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-KP8Zco\""
   ],
   "metadata": {
    "id": "o7oDiePZJfwe"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "TODO: 这里Haruhi_first_merge_res.jsonl本来就是synthesized的，\n",
    "\n",
    "最好替换为之前的输出 ./output/chat_from_dialogue.jsonl"
   ],
   "metadata": {
    "id": "LXJz1nQ3ItMK"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "!python synthesis_chat.py -input ./data/Haruhi_first_merge_res.jsonl\\\n",
    "                         -output ./output/chat_augmented_by_foo.jsonl \\\n",
    "                         -method foo"
   ],
   "metadata": {
    "id": "a-GTiO1xIOie"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## chat2dialogue 通过inference引擎，将第一句话输入到里面去要出更多的对话\n",
    "#### - role_name 要和 src_reform/config.ini 中的角色名字一致\n",
    "#### - output_dialogue  是可选项，默认输出文件和 input_chat 在同一目录\n",
    "#### - other_names 可选"
   ],
   "metadata": {
    "id": "1ycjz0XmJZWj"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-Z2fFp\""
   ],
   "metadata": {
    "id": "bSfBcR28CH_M"
   },
   "execution_count": 10,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "!python chat2dialogue.py -input_chat ./data/test_query_chat_Haruhi.jsonl\\\n",
    "                        -output_dialogue ./output/haruhi_output_dialogue.jsonl\\\n",
    "                        -role_name \"凉宫春日\" \\\n",
    "                        -other_names 凉宫 凉宫春日"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hdNTuVtzI2Xu",
    "outputId": "fd3447f2-2a76-459c-f96f-9ceaf2a43ce9"
   },
   "execution_count": 19,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "正在下载Luotuo-Bert\n",
      "Downloading (…)lve/main/config.json: 100% 35.9k/35.9k [00:00<00:00, 101MB/s]\n",
      "Downloading (…)solve/main/models.py: 100% 21.1k/21.1k [00:00<00:00, 24.8MB/s]\n",
      "A new version of the following files was downloaded from https://huggingface.co/silk-road/luotuo-bert-medium:\n",
      "- models.py\n",
      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
      "Downloading pytorch_model.bin: 100% 1.31G/1.31G [00:15<00:00, 84.7MB/s]\n",
      "Luotuo-Bert下载完毕\n",
      "<bound method RawConfigParser.items of <configparser.ConfigParser object at 0x7f7776b5ffd0>>\n",
      "正在下载Luotuo-Bert\n",
      "Luotuo-Bert下载完毕\n",
      "Traceback (most recent call last):\n",
      "  File \"/content/Chat-Haruhi-Suzumiya/kyon_generator/chat2dialogue.py\", line 126, in <module>\n",
      "    main(input_chat, output_dialogue, role_name, other_names_lis)\n",
      "  File \"/content/Chat-Haruhi-Suzumiya/kyon_generator/chat2dialogue.py\", line 100, in main\n",
      "    chatgpt.set_training(role_name, other_names.split())\n",
      "AttributeError: 'list' object has no attribute 'split'\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## generate_character 将台本文件保存成jsonl文件，创建新的角色\n",
    "#### 如果是复写台本 jsonl 则不需要指定prompt"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html\n",
      "Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\19644\\PycharmProjects\\Chat-Haruhi-Suzumiya\\kyon_generator\\generate_character.py\", line 8, in <module>\n",
      "    from src_reform import utils, checkCharacter\n",
      "  File \"C:\\Users\\19644\\PycharmProjects\\Chat-Haruhi-Suzumiya\\src_reform\\utils.py\", line 2, in <module>\n",
      "    from transformers import AutoModel, AutoTokenizer\n",
      "ImportError: cannot import name 'AutoModel' from 'transformers' (C:\\ProgramData\\anaconda3\\Lib\\site-packages\\transformers\\__init__.py)\n"
     ]
    }
   ],
   "source": [
    "!python generate_character.py -cn_role_name 坤坤\\\n",
    "                              -en_role_name kunkun\\\n",
    "                              -text_folder ../characters/kunkun/texts"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-04T23:46:50.980992700Z",
     "start_time": "2023-08-04T23:46:48.409310500Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ]
}
